HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN

Spatiotemporal representations learned using 3D convolutional neural networks (CNN) are currently used in state-of-the-art approaches for action-related tasks. However, 3D-CNN notorious being memory and compute resource intensive as compared with more simple 2D-CNN architectures. We propose to hallucinate spatiotemporal from a teacher student. By requiring the predict future intuit upcoming activity, it is encouraged gain deeper understanding of actions how they evolve. The hallucination task treated an auxiliary task, which can be any other multitask learning setting. Thorough experimental evaluation, shown that indeed helps improve performance on action recognition, quality assessment, dynamic scene recognition From practical standpoint, able without actual enable deployment resource-constrained scenarios, such limited computing power and/or lower bandwidth. also observed our has utility not only during training phase, but pre-training phase.